Image forming apparatus, image forming method, image forming system, and recording medium

ABSTRACT

An image forming apparatus includes an imager that is electrically connected to an image sensor disposed at a position where light that has passed through a sample slice is incident on the image sensor, and an illumination system that emits illumination light successively in different illumination directions relative to a sample slice to illuminate the sample slice with the illumination light and that emits a first light having a peak in a first wavelength range and a second light having a peak in a second wavelength range. The image forming apparatus obtains a plurality of first-color images with the image sensor while the sample slice is being illuminated with the first light serving as the illumination light successively in the different illumination directions. The image forming apparatus obtains at least one second-color image with the image sensor while the sample slice is being illuminated with the second light in at least one of the different illumination directions. The image forming apparatus generates a high-resolution image on the basis of the plurality of first-color images and the at least one second-color image.

BACKGROUND 1. Technical Field

The present disclosure relates to image forming apparatuses, imageforming methods, image forming systems, and recording media.

2. Description of the Related Art

In a pathological diagnosis, tissue is excised from an internal organ ora tumor and examined to make a definite diagnosis of disease or todetermine the spread of a lesion. The excised tissue section is thensliced to a thickness of several microns so that a tissue slice can beobserved under a microscope, and a pathology slide (specimen) is createdby placing the tissue slice between glass plates. A pathologicaldiagnosis is made to determine whether the cancer is benign ormalignant, and thus the number of specimens created per day forpathological diagnoses at a hospital is as high as several hundred.Unlike radiographic images or the like, pathological specimens cannot besaved in the form of electronic data. Therefore, it is typical topreserve the specimens so that the created specimens can be examined ata later time.

Conventionally, microscopes are used to observe microstructures, such asbiological tissue. A microscope magnifies light that has passed througha subject or light reflected by a subject through a lens. An examinerdirectly observes an image formed by the magnified light. If a digitalmicroscope that captures a microscopy image with a camera and displaysthe image on a display is used, multiple people can observe the image atthe same time or observe the image at remote locations. The camera isdisposed at a focal point of the microscope and captures the image thathas been magnified by a lens of the microscope.

Japanese Unexamined Patent Application Publication No. 4-316478discloses a technique for observing a microstructure through a contactimage sensing (CIS) method. In the CIS method, a subject is placeddirectly on an image sensor, and an image of the subject is captured.The image is not magnified by a lens, and thus the pixel size of theimage sensor determines the resolution. In other words, the smallerpixel size enables a more detailed image of a microstructure to becaptured.

As stated above, when an image is captured by using a conventional CISmethod, a resolution that exceeds the resolution determined by the pixelsize of the image sensor cannot be achieved.

SUMMARY

One non-limiting and exemplary embodiment provides an image formingapparatus that can achieve a resolution that exceeds the resolutiondetermined by the pixel size of an image sensor.

In one general aspect, the techniques disclosed here feature an imageforming apparatus that includes an imager that is electrically connectedto an image sensor disposed such that light that has passed through asample slice is incident thereon, an illumination system that emitsillumination light successively in different illumination directionsrelative to the sample slice to illuminate the sample slice with theillumination light and that emits a first light having a peak in a firstwavelength range and a second light having a peak in a second wavelengthrange, a controller that is connected to the imager and to theillumination system and that controls the imager and the illuminationsystem, and an image processor that obtains data of a plurality ofimages from the image sensor and combines the plurality of images togenerate a high-resolution image of the sample slice that has aresolution higher than a resolution of each of the plurality of images.The controller obtains a plurality of first-color images with the imagesensor while the sample slice is being illuminated with the first lightserving as the illumination light successively in the differentillumination directions, and obtains at least one second-color imagewith the image sensor while the sample slice is being illuminated withthe second light serving as the illumination light in at least one ofthe different illumination directions. The image processor generates thehigh-resolution image on the basis of the plurality of first-colorimages and the at least one second-color image.

According to the present disclosure, a microscope that does not includea lens can be provided, and thus space-savings and cost-savings can beachieved.

It should be noted that general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a computer-readable storage medium, or any selectivecombination thereof. A computer-readable storage medium includes, forexample, a non-transitory storage medium, such as a compact-discread-only memory (CD-ROM).

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary method for creating a prepared specimenfor a pathological diagnosis;

FIG. 2 schematically illustrates a section of the prepared specimenbeing observed under a microscope;

FIG. 3 is an illustration for describing a principle of a CISobservation method;

FIG. 4 illustrates an exemplary method for creating a prepared specimenaccording to a first embodiment of the present disclosure;

FIG. 5 schematically illustrates an exemplary configuration, along asection, of a prepared specimen that includes an image sensor and apackage according to the first embodiment;

FIG. 6 schematically illustrates an exemplary configuration, along asection, of the prepared specimen and a socket;

FIG. 7 schematically illustrates another exemplary configuration, alonga section, of the prepared specimen and the socket;

FIG. 8 illustrates an exemplary configuration of an image formingapparatus according to the first embodiment;

FIG. 9 is a plan view illustrating an example of an illumination systemprovided in the image forming apparatus according to the firstembodiment;

FIG. 10 is a perspective view illustrating a relationship between theillumination system illustrated in FIG. 9 and an image sensor;

FIG. 11 is a plan view illustrating a detailed configuration of theillumination system illustrated in FIG. 9;

FIG. 12 is a plan view illustrating another example of the illuminationsystem provided in the image forming apparatus according to the firstembodiment;

FIG. 13 is a perspective view illustrating a relationship between theillumination system illustrated in FIG. 12 and an image sensor;

FIG. 14A illustrates an exemplary operation of the image formingapparatus according to the first embodiment;

FIG. 14B illustrates the exemplary operation of the image formingapparatus according to the first embodiment;

FIG. 14C illustrates the exemplary operation of the image formingapparatus according to the first embodiment;

FIG. 15A illustrates another exemplary operation of the image formingapparatus according to the first embodiment;

FIG. 15B illustrates the other exemplary operation of the image formingapparatus according to the first embodiment;

FIG. 16A is a sectional view schematically illustrating an operation ofcapturing, with an image sensor, an image of a subject that is beingilluminated with illumination light at a given angle, according to thefirst embodiment;

FIG. 16B is a plan view illustrating an exemplary ratio of the area of aphotodiode (PD) to the area of a pixel;

FIG. 17A is a plan view schematically illustrating an arrangement ofpixels in the image obtained through the operation illustrated in FIG.16A;

FIG. 17B schematically illustrates a biological image obtained in theillumination state illustrated in FIG. 16A;

FIG. 18 is a sectional view schematically illustrating an operation ofcapturing, with an image sensor, an image of a subject that is beingilluminated with illumination light at another angle, according to thefirst embodiment;

FIG. 19A is a plan view schematically illustrating an arrangement ofpixels in the image obtained through the operation illustrated in FIG.18;

FIG. 19B schematically illustrates a biological image obtained in theillumination state illustrated in FIG. 18;

FIG. 20 is a sectional view schematically illustrating an operation ofcapturing, with an image sensor, an image of a subject that is beingilluminated with illumination light at yet another angle, according tothe first embodiment;

FIG. 21A is a plan view schematically illustrating an arrangement ofpixels in the image obtained through the operation illustrated in FIG.20;

FIG. 21B schematically illustrates a biological image obtained in theillumination state illustrated in FIG. 20;

FIG. 22 is a sectional view schematically illustrating an operation ofcapturing, with an image sensor, an image of a subject that is beingilluminated with illumination light at yet another angle, according tothe first embodiment;

FIG. 23A is a plan view schematically illustrating an arrangement ofpixels in the image obtained through the operation illustrated in FIG.22;

FIG. 23B schematically illustrates a biological image obtained in theillumination state illustrated in FIG. 22;

FIG. 24A is a block diagram schematically illustrating an exemplaryconfiguration according to the first embodiment;

FIG. 24B illustrates an exemplary process for generating a combinedimage;

FIG. 25A illustrates an image according to a comparative example;

FIG. 25B illustrates an exemplary high-resolution image obtained throughimage processing according to the first embodiment;

FIG. 25C illustrates another image according to the comparative example;

FIG. 26 is a timing chart illustrating an exemplary operation accordingto the first embodiment;

FIG. 27 is a schematic diagram illustrating an exemplary configurationof an image forming system according to a second embodiment of thepresent disclosure;

FIG. 28 is a schematic diagram illustrating an exemplary illuminationsystem that includes an illumination angle adjustment mechanism forchanging the posture of a socket;

FIG. 29 is a schematic diagram illustrating an exemplary configurationof an illumination system that includes an illumination angle adjusterfor adjusting the angle at which illumination light is incident on asample slice; and

FIG. 30 is a schematic diagram illustrating another exemplaryconfiguration of the illumination system that includes the illuminationangle adjuster for adjusting the angle at which illumination light isincident on a sample slice.

DETAILED DESCRIPTION

In the medical field, microscopes are used to observe cells. Observationof cell shape can make it possible to determine whether a patient has adisease; and if the patient has a disease, the benignity or the degreeof malignancy of the disease can be determined. In a type of diagnosiscalled a pathological diagnosis, a specimen taken from a patient issliced to a thickness of approximately 4 μm to observe the cellstherein. The cells are translucent, and a microscopy image thereof haslow contrast. Therefore, the cells are subjected to staining so that thestructure of the cells can be seen more easily.

With reference to FIG. 1, an exemplary method for creating a preparedspecimen A01 for a pathological diagnosis is described.

As illustrated in FIG. 1, a slice A02 is placed on a slide (transparentplate) A03. The slide A03 typically has a thickness of 1 mm, a length of76 mm, and a width of 26 mm. The slice A02, along with the slide A03, isimmersed in a stain solution A04 and is thus stained. Upon staining theslice A02, the slice A02 turns into a sample slice (hereinafter, alsoreferred to as a stained slice) A05. A mounting medium A06 is applied tothe slide A03 in order to protect and secure the stained slice A05.Thereafter, a cover slip A07 is placed, and the prepared specimen A01 isthus completed.

FIG. 2 schematically illustrates a section of the prepared specimen A01being observed under a microscope.

As illustrated in FIG. 2, the stained slice A05 is placed on the slideA03. The cover slip A07 is fixed to the slide A03 with the mountingmedium A06 provided therebetween. The stained slice A05 is surrounded bythe mounting medium A06 such that it is located between the cover slipA07 and the slide A03.

When the prepared specimen A01 is placed under an optical microscope forobservation, the prepared specimen A01 is illuminated, at a lower sidethereof, with illumination light G02 emitted by a light source G01. Theillumination light G02 passes through the slide A03, the stained sliceA05, the mounting medium A06, and the cover slip A07 and is incident onan objective lens G03 of the microscope.

When the prepared specimen A01 is observed under such an opticalmicroscope, there arises a problem in that it takes time to set themagnification or observation area.

Subsequently, a principle of a CIS observation method is described withreference to FIG. 3.

A prepared specimen E01 illustrated in FIG. 3 includes an image sensorB01, in place of the cover slip A07. The prepared specimen E01 includesa transparent plate (in this example, a slide) A03, the image sensorB01, and a stained slice (subject) A05, which is surrounded by themounting medium A06. The image sensor B01 is fixed to the slide A03 withthe mounting medium A06 provided therebetween. As the image sensor B01,a solid-state image sensor may be used in which photoelectric convertersare arrayed in a matrix on an imaging surface. Each photoelectricconverter is typically a photodiode provided on a semiconductor layer oron a semiconductor substrate. The photoelectric converters receiveincident light and generate electric charge. The resolution of atwo-dimensional image sensor is dependent on the array pitch or thearray density of photoelectric converters arrayed on the imagingsurface. In recent years, the array pitch of photoelectric convertershas been reduced to be equal to visible-light wavelengths. A typicalexample of the image sensor B01 is a charge-coupled device (CCD) imagesensor or a metal-oxide semiconductor (MOS) image sensor.

When an image is to be captured, the illumination light G02 passesthrough the slide A03, the stained slice A05, and the mounting mediumA06, and reaches the image sensor B01 in the prepared specimen E01. Theimage sensor B01 is electrically connected to circuitry (notillustrated) and carries out an imaging operation. The image sensor B01captures an image of the stained slice A05 and outputs an image signalcorresponding to an optical transmittance distribution (densitydistribution) of the stained slice A05. Consequently, an image of thestained slice A05 is obtained.

According to such a CIS observation method, an optical system, such as alens, is not present between the element that captures an image and thestained slice A05 (subject). Nonetheless, as minute photoelectricconverters (photodiodes) are arrayed at high density on the imagingsurface of the image sensor B01, an image showing the fine structure ofthe stained slice A05 can be obtained. Hereinafter, the resolution isbriefly described.

The resolution of the optical microscope described above is defined by atwo-point resolution. The resolution δ of two point light sources areexpressed through the following expression (1) in accordance with theRayleigh criterion.

$\begin{matrix}{\delta = \frac{0.61 \times \lambda}{NA}} & (1)\end{matrix}$

Here, λ represents the wavelength of light, and NA represents thenumerical aperture of an objective lens.

For example, when the numerical aperture NA of the objective lens is0.25 and the wavelength λ is 555 nm, the resolution δ is 1.35 μm throughthe expression (1). To achieve a resolution equivalent to the resolutionδ through the CIS method, the pixel pitch of an image sensor to be usedmay be set to 1.35 μm. If an image is to be captured with a resolutionthat is twice the aforementioned resolution, the pixel pitch of theimage sensor may be reduced by one-half to approximately 0.6 μm.However, it is difficult to further miniaturize the pixel structure ofan image sensor, and such miniaturization leads to an increase infabrication cost.

According to an embodiment of the present disclosure, a subject isilluminated with light in multiple illumination directions relative toan image sensor, and multiple images are obtained by light that haspassed through an area smaller than the size of a pixel. The obtainedimages are then combined to increase the resolution.

Typically, a color image sensor that includes a color mosaic filter isused to obtain a color image. However, the period at which pixels of thesame color are arrayed in such an image sensor is extended, and thus theresolution decreases. To suppress such a decrease in the resolution, amonochrome image sensor that does not include a color mosaic filter maybe used. Then, while a sample slice is illuminated with illuminationlight of different colors, such as red (R), green (G), and blue (B), ina time sequential manner, images may be captured under the illuminationlight of the respective colors. Through this, for example, a red (R)image, a green (G) image, and a blue (B) image are obtained. Theseimages can then be combined to form a color image.

However, if, for example, the red (R) image, the green (G) image, andthe blue (B) image are to be obtained for all illumination directions,the number of instances of imaging increases, and the amount of data ofthe obtained images becomes huge. According to an embodiment of thepresent disclosure, although such an increase in the amount of data isbeing suppressed, a high-resolution image can be obtained by employingconfigurations described hereinafter.

An overview of an aspect of the present disclosure is as follows.

An image forming apparatus according to one embodiment of the presentdisclosure includes an imager that is electrically connected to an imagesensor disposed at a position where light that has passed through asample slice is incident thereon, an illumination system that emitsillumination light successively in different illumination directionsrelative to the sample slice and illuminates the sample slice with theillumination light, a controller that is connected to the imager and tothe illumination system and that controls the imager and theillumination system, and an image processor that obtains data of aplurality of images from the image sensor and combines the plurality ofimages to generate a high-resolution image of the sample slice that hasa resolution higher than a resolution of each of the plurality ofimages. The illumination system emits a first light having a peak in afirst wavelength range and a second light having a peak in a secondwavelength range. The controller obtains a plurality of first-colorimages with the image sensor while the sample slice is being illuminatedwith the first light serving as the illumination light successively inthe different illumination directions. In addition, the controllerobtains at least one second-color image with the image sensor while thesample slice is being illuminated with the second light serving as theillumination light in at least one of the different illuminationdirections. The image processor generates the high-resolution image onthe basis of the plurality of first-color images and the at least onesecond-color image.

In one embodiment, the imager releasably supports a prepared specimenthat includes the sample slice and the image sensor, and is electricallyconnected to the image sensor in a state in which the imager supportsthe prepared specimen.

In one embodiment, the illumination system illuminates the sample slicewith the first light emitted in at least four different illuminationdirections. The image sensor obtains at least four different first-colorimages while the sample slice is being illuminated with the first light,and each of the at least four different first-color images is an imageof a different portion of the sample slice. The image processorgenerates a high-resolution image of the sample slice on the basis ofthe at least four different first-color images.

In one embodiment, the illumination system emits a third light having apeak in a third wavelength range. The controller obtains at least onethird-color image with the image sensor while the sample slice is beingilluminated with the third light serving as the illumination light in atleast one of the different illumination directions.

In one embodiment, the first light has a wavelength in a range of 495 nmto 570 nm inclusive. The second light has one of a wavelength in a rangeof 620 nm to 750 nm inclusive and a wavelength in a range of no lessthan 450 nm to less than 495 nm.

In one embodiment, the first light has a wavelength in a range of 495 nmto 570 nm inclusive. The second light has a wavelength in a range of 620nm to 750 nm inclusive. The third light has a wavelength in a range ofno less than 450 nm to less than 495 nm.

In one embodiment, the illumination system includes a light source thatemits the illumination light, and the light source is moved successivelyto different positions corresponding to the respective differentillumination directions.

In one embodiment, the illumination system includes a plurality of lightsources that successively emit the illumination light, and the lightsources are disposed at different positions corresponding to therespective different illumination directions.

In one embodiment, the illumination system includes a mechanism thatchanges at least one of positions and directions of the sample slice andthe image sensor.

An image forming apparatus according to one embodiment includes anillumination angle adjuster that adjusts an angle at which theillumination light is incident on the sample slice. The illuminationangle adjuster adjusts the angle at which the illumination light isincident on the sample slice in such a manner that the illuminationlight emitted successively in the different illumination directions bythe illumination system passes through different portions of the sampleslice and is incident on photoelectric converters of the image sensor.

An image forming method according to one embodiment of the presentdisclosure includes emitting illumination light successively indifferent illumination directions relative to a sample slice andilluminating the sample slice with the illumination light, obtaining aplurality of images corresponding to the respective illuminationdirections with an image sensor disposed at a position where light thathas passed through the sample slice is incident thereon, and combiningthe plurality of images to generate a high-resolution image of thesample slice that has a resolution higher than a resolution of each ofthe plurality of images. The illuminating of the sample slice with theillumination light includes illuminating the sample slice with a firstlight having a peak in a first wavelength range in the differentillumination directions and illuminating the sample slice with a secondlight having a peak in a second wavelength range in at least one of thedifferent illumination directions. The obtaining of the plurality ofimages includes obtaining a plurality of first-color images while thesample slice is being illuminated with the first light serving as theillumination light successively in the different illuminationdirections, and obtaining at least one second-color image while thesample slice is being illuminated with the second light serving as theillumination light in at least one of the different illuminationdirections.

An image forming system according to one embodiment of the presentdisclosure includes an imager that is electrically connected to an imagesensor disposed at a position where light that has passed through asample slice is incident thereon, an illumination system that emitsillumination light successively in different illumination directionsrelative to the sample slice and illuminates the sample slice with theillumination light, and a computer. The illumination system emits afirst light having a peak in a first wavelength range and a second lighthaving a peak in a second wavelength range. The computer executesobtaining a plurality of first-color images with the image sensor whilethe sample slice is being illuminated with the first light serving asthe illumination light successively in the different illuminationdirections, obtaining at least one second-color image with the imagesensor while the sample slice is being illuminated with the second lightserving as the illumination light in at least one of the differentillumination directions, and generating such a high-resolution image onthe basis of the plurality of first-color images and the at least onesecond-color image that has a resolution higher than a resolution ofeach of the first-color images.

A recording medium according to one embodiment of the present disclosureis a non-transitory computer-readable recording medium storing acomputer program to be used in the image forming system described above.The computer program causes the image forming system to executeobtaining a plurality of first-color images with the image sensor whilethe sample slice is being illuminated with the first light serving asthe illumination light successively in the different illuminationdirections, obtaining at least one second-color image with the imagesensor while the sample slice is being illuminated with the second lightserving as the illumination light in at least one of the differentillumination directions, and generating an image on the basis of theplurality of first-color images and the at least one second-color imagethat has a resolution higher than a resolution of each of thefirst-color images.

An image forming method according to the present disclosure includesemitting a first light having a peak in a first wavelength range in afirst illumination direction, the first light emitted in the firstillumination direction passing thorough a first portion of a sample toemit a first resulting light from the first portion; obtaining firstdirection data showing that the first light is emitted in the firstillumination direction; receiving the first resulting light on an imagesensor; outputting first image data based on the first resulting lightreceived by the image sensor; emitting a second light having a peak in asecond wavelength range in the first illumination direction, the secondlight emitted in the first illumination direction passing through thefirst portion of the sample to emit a second resulting light from thefirst portion; obtaining second direction data showing that the secondlight is emitted in the first illumination direction; receiving thesecond resulting light on the image sensor; outputting second image databased on the second resulting light received by the image sensor;emitting the first light in a second illumination direction, the firstlight emitted in the second illumination direction passing through asecond portion of the sample to emit a third resulting light from thesecond portion; obtaining third direction data showing that the firstlight is emitted in the second illumination direction; receiving thethird resulting light on the image sensor; outputting third image databased on the third resulting light received by the image sensor; andpreparing a first image based on the first data, the second data, thethird data, the first illumination direction, the second illuminationdirection, and the third illumination direction. In the image formingmethod, the second light is not emitted in the second direction.

An image forming apparatus according to an embodiment of the presentdisclosure includes an imager that releasably supports a preparedspecimen that includes a sample slice and an image sensor disposed suchthat light that has passed through the sample slice is incident on theimage sensor. A point to be noted here is that the image sensor forms apart of the prepared specimen. Such a prepared specimen may be called anelectronic prepared specimen. The imager is electrically connected tothe image sensor in a state in which the imager supports such a preparedspecimen.

An image forming apparatus according to an embodiment of the presentdisclosure includes an illumination system that emits illumination lightsuccessively in different illumination directions relative to a sampleslice and illuminates the sample slice with the illumination light. Theillumination system is configured to emit a first light having a peak ina first wavelength range and a second light having a peak in a secondwavelength range. In addition, the illumination system may be configuredto emit a third light having a peak in a third wavelength range. Thefirst light, for example, may have a wavelength in a range of 495 nm to570 nm inclusive. The second light and the third light may have awavelength in a range of 620 nm to 750 nm inclusive and a wavelength ina range of no less than 450 nm to less than 495 nm, respectively. Forsimplifying the description, light having a wavelength in a range of 495nm to 570 nm inclusive may be referred to as green light, light having awavelength in a range of 620 nm to 750 nm inclusive may be referred toas red light, and light having a wavelength in a range of no less than450 nm to less than 495 nm may be referred to as blue light.

The specific configuration of the illumination system is not limited aslong as the illumination system has a function to change the angle(illumination angle) of the illumination light incident on a sampleslice. The illumination system may include one or both of a mechanismfor moving a light source and a mechanism (e.g., gonio-mechanism) formoving a sample slice in order to change the illumination angle.

The image forming apparatus according to the present disclosure furtherincludes a controller and an image processor. The controller isconnected to the imager and to the illumination system and controls theimager and the illumination system. The image processor obtains data ofa plurality of images from the image sensor in the prepared specimensupported by the imager and combines the plurality of images to generatea high-resolution image of the sample slice that has a resolution higherthan a resolution of each of the plurality of images.

The controller is configured to obtain a plurality of first-color imageswith the image sensor while the sample slice is being illuminated withthe first light serving as the illumination light successively in thedifferent illumination directions. In addition, the controller isconfigured to obtain at least one second-color image with the imagesensor while the sample slice is being illuminated with the second lightserving as the illumination light in at least one of the differentillumination directions. In one specific example, the first-color imagemay be an image obtained while the sample is being illuminated with thegreen light serving as the illumination light, and the second-colorimage may be an image obtained while the sample is being illuminatedwith the red light serving as the illumination light. The sample sliceis typically stained a specific color, and even a color image of such asample slice that is expressed through a mixture of two primary colorsmay include sufficiently useful information.

In addition, in a case in which the illumination system can emit thethird light, the controller may be configured to obtain at least onethird-color image with the image sensor while the sample slice is beingilluminated with the third light serving as the illumination light in atleast one of the different illumination directions. A color imageexpressed by a mixture of three primary colors can reproduce colorinformation of an image to be visually perceived by the human eye at ahigh level and is suitable when a medical practitioner makes adiagnosis.

The image processor according to the present disclosure is configured togenerate an image on the basis of the plurality of first-color imagesand the at least one second-color image that has a resolution higherthan a resolution of each of the first-color images. As it becomes clearfrom the following description, an image generated by combining aplurality of first-color images obtained with the image sensor while thesample slice is being illuminated with the first light serving as theillumination light successively in the different illumination directionsresults in an image having a relatively high resolution. Meanwhile, atleast one second-color image obtained with the image sensor while thesample slice is being illuminated with the second light serving as theillumination light in at least one of the different illuminationdirections results in an image having a relatively low resolution. Evenif the resolution of some of the three primary color images is low, itis possible to generate a high-resolution combined image, and the imageprocessor according to the present disclosure has been conceived of bypaying attention to that feature.

Hereinafter, embodiments of the present disclosure are described indetail with reference to the drawings.

It is to be noted that the embodiments described hereinafter illustrategeneral or specific examples. The Numerical values, the shapes, thematerials, the components, the arrangements, the positions, and theconnection modes of the components, the steps, the order of the steps,and so forth indicated in the embodiments hereinafter are examples, andare not intended to limit the present disclosure. Furthermore, among thecomponents in the embodiments hereinafter, a component that is notdescribed in an independent claim indicating the broadest concept isconsidered to be an optional component.

It is to be noted that these general or specific embodiments may beimplemented as a system, a method, an integrated circuit, a computerprogram, a computer-readable storage medium, or any selectivecombination thereof.

First Embodiment

With reference to FIG. 4, an exemplary method for creating a preparedspecimen according to an embodiment of the present disclosure isdescribed.

As illustrated in FIG. 4, a slice A02 is placed on a slide (transparentplate) A03. The slice A02, along with the slide A03, is immersed in astain solution A04 and is thus stained. Upon staining the slice A02, theslice A02 becomes a stained slice A05. A mounting medium A06 is appliedto the slide A03 in order to protect and secure the stained slice A05.Thereafter, in place of the cover slip A07 illustrated in FIG. 1, animage sensor B01 is placed. In the example illustrated in FIG. 4, theimage sensor B01 is connected, at its rear side, to a package 12. Thus,a prepared specimen 11 is completed.

FIG. 5 schematically illustrates an exemplary configuration, along asection, of the prepared specimen 11, which includes the image sensorB01 and the package 12. In the example illustrated in FIG. 5, the imagesensor B01 is contained in the package 12. The image sensor B01 and thepackage 12 are electrically interconnected through wire electrodes(bonding wires) F01. The electrical connection between the image sensorB01 and the package 12 is not limited to the example illustrated in FIG.5, and the electrode F01 does not need to be wire-shaped. The package 12illustrated in FIG. 5 includes a base and walls (side walls) that form aspace in which the image sensor B01 is contained. The configuration ofthe image sensor B01 and the package 12 may be similar to theconfiguration of a known image sensor package. In the exampleillustrated in FIG. 5, the width (a dimension in the horizontaldirection along the drawing plane) of the slide A03 is narrower than thewidth of the package 12. Alternatively, the width of the slide A03 maybe greater than the width of the package 12.

FIG. 6 schematically illustrates the prepared specimen 11, which can beused in the embodiments of the present disclosure, and a portion of animager, which releasably supports the prepared specimen 11. The overallconfiguration of the imager is described later. The imager includes asocket C03 configured to load the prepared specimen 11. The socket C03is electrically connected to a circuit board C05. The electricalconnection between the socket C03 and the circuit board C05 may beachieved, for example, as terminals provided on a rear side of thesocket C03 make contact with wires or electrode pads provided on thecircuit board C05. The circuit board C05 may have a known configuration,and may be, for example, a multilayer printed-circuit board. The socketC03 may be mounted on the circuit board C05 through any known method formounting an electronic component on a circuit board. Terminals 13 areprovided on a rear side of the package 12 for electrically connectingthe image sensor B01 to external circuitry. The socket C03 includesterminals C04 arranged so as to be electrically connected to therespective terminals 13 provided on the package 12.

FIG. 7 schematically illustrates an exemplary configuration in which theprepared specimen 11 is loaded on the socket C03. The prepared specimen11 is tentatively fixed to the socket C03 through the socket C03 itselfor through another mechanism. As the prepared specimen 11 is loaded onthe socket C03, the terminals C04 on the socket C03 are electricallyconnected to the image sensor B01 through the terminals 13 on thepackage 12. The configuration of the socket C03 is not limited to thisexample. The electrical connection between the socket C03 and the imagesensor B01 is not limited to the above example, either.

In the state illustrated in FIG. 7, the prepared specimen 11 isilluminated with illumination light from the upper side, andillumination light that has passed through the stained slice A05 isincident on the image sensor B01. In this manner, necessary images arecaptured in multiple instances. Upon imaging of a target preparedspecimen 11 being completed, the prepared specimen 11 is removed fromthe socket C03. Then, a subsequent target prepared specimen 11 is loadedon the socket C03.

FIG. 8 schematically illustrates an exemplary configuration of an imageforming apparatus 10 according to the present embodiment.

The image forming apparatus 10 illustrated in FIG. 8 includes anillumination system C09. The illumination system C09 causes light to beincident on the image sensor B01 through the slide A03 provided in theprepared specimen 11, which is loaded on the socket C03. Theconfiguration and the operation of the illumination system C09 aredescribed later. In the example illustrated in FIG. 8, the illuminationsystem C09 is located above the prepared specimen 11 that is supportedby an imager 90. The embodiment of the present disclosure, however, isnot limited to such an example. The vertical relationship between theillumination system C09 and the prepared specimen 11 may be reversed, ora line connecting the illumination system C09 and the prepared specimen11 may be at an angle relative to the vertical direction.

In the example illustrated in FIG. 8, the image forming apparatus 10includes a control device (computer) C06. The computer C06 may becircuitry, such as a processor, mounted in a device or may be anindependent device. In other words, the computer C06 may be a devicethat is separate from the imager 90 and/or the illumination system C09.In the exemplary configuration illustrated in FIG. 8, the computer C06includes a controller 120, an image processor 140, and a memory 145. Thecontroller 120 is configured to control the image sensor B01 in theprepared specimen 11, which is loaded on the socket C03, and theillumination system C09, and to thus cause the image sensor B01 tocapture an image of a stained slice in the prepared specimen 11.

As described with reference to FIG. 7, the package 12 becomeselectrically connected to the socket C03, upon being located on thesocket C03. The socket C03 is connected to the computer C06 illustratedin FIG. 8 through the circuit board C05 illustrated in FIG. 7.

Image data obtained through imaging is subjected to combining and pixelinterpolation processing by the image processor 140. Through suchprocessing, a higher-resolution image of the stained slice is generated.The generated image, for example, may be displayed in a display C07 andstored in the memory 145 or a database 148.

FIG. 9 is a plan view schematically illustrating an exemplaryarrangement of light source elements in the illumination system C09,which can be used in the embodiments of the present disclosure. In theexample illustrated in FIG. 9, 25 light source elements 20 are arrayedin a matrix. Here, the light source elements 20 are arrayed in a matrixof 5 by 5, which is provided on a light-emission surface of theillumination system C09.

As illustrated in FIG. 10, the illumination system C09 that includes thelight source elements 20 arrayed in a matrix can cause illuminationlight to be incident on the image sensor B01 provided in the preparedspecimen 11 at different angles. The illumination light emitted by thelight source elements 20 is substantially parallel light at the imagesensor B01. For example, when the illumination system C09 includes atleast four light source elements 20, the illumination system C09 cancause the illumination light to be incident on the image sensor B01provided in the prepared specimen 11 successively in at least fourdifferent directions. Each of the light source elements 20 in theillumination system C09 may be an element constituted by a combinationof a light-emitting element, such as a light-emitting diode (LED), and acolor filter. In addition, each of the light source elements 20 may beprovided with an optical element, a reflection mirror, or the like foradjusting the divergence of the light rays.

In the present embodiment, as illustrated in FIG. 11, a light sourceelement 20 a located at the center of the light-emission surface of theillumination system C09 may be constituted by three LED chips 20G, 20R,and 20B that have peaks in mutually different wavelength ranges. Thethree LED chips 20G, 20R, and 20B are configured to emit, respectively,a first light having a peak in a first wavelength range, a second lighthaving a peak in a second wavelength range, and a third light having apeak in a third wavelength range. In the present embodiment, the firstlight, the second light, and the third light are green light, red light,and blue light, respectively. The light source element 20 a does notneed to be constituted by a plurality of LED chips, and may instead beconstituted by a single white LED chip. In addition, the light sourceelement 20 a may be a discharge pipe or a laser element.

In the present embodiment, 24 light source elements 20 b excluding thelight source element 20 a located at the center are configured to emitthe first light, namely, the green light in this example. The lightsource elements 20 b typically include the LED chips 20G.

The illumination system C09 configured as described above can illuminatea sample slice in the prepared specimen 11 loaded on the socket C03 withthe illumination light emitted successively in 25 different illuminationdirections with the sample slice serving as a reference. Theillumination system C09 can emit the first light (green light in thisexample) having a peak in the first wavelength range in 25 differentillumination directions. In this example, the illumination system C09 isconfigured to emit the second light (red light in this example) having apeak in the second wavelength range and the third light (blue light inthis example) having a peak in the third wavelength range in a singleillumination direction.

FIG. 12 illustrates another exemplary configuration of the illuminationsystem C09. In this example, a single light source element 22 moves soas to emit light at different light source positions 24. The movablelight source element 22 can emit the first light (green light in thisexample) having a peak in the first wavelength range, the second light(red light in this example) having a peak in the second wavelengthrange, and the third light (blue light in this example) having a peak inthe third wavelength range selectively or simultaneously. Such a lightsource element 22, for example, may have a configuration similar to theconfiguration of the light source element 20 a located at the center ofthe illumination system C09 illustrated in FIG. 11.

Any desired mechanism can be employed to move the light source element22. For example, the light source element 22 can be made to emit lightat a desired position with the use of two stepping motors configured tomove a movable portion in the directions of the X-axis and the Y-axis.

As illustrated in FIG. 13, the illumination system C09 that includes themovable light source element 22 can cause the illumination light to beincident on the image sensor B01 in the prepared specimen 11 atdifferent angles. The illumination light emitted by the light sourceelement 22 is substantially parallel light at the image sensor B01. Thelight source element 22 may be an element constituted by a combinationof a light-emitting element and a color filter. In addition, the lightsource element 22 may be provided with an optical element, a reflectionmirror, or the like for adjusting the divergence of the light rays.

When the illumination system C09 illustrated in FIG. 12 is employed, theillumination system C09, for example, causes the illumination light ofthree different colors to be successively incident on the image sensorB01 from a light source position 24 c located at the center of theillumination system C09. The illumination system C09 causes theillumination light of a single color to be incident on the image sensorB01 from the remaining light source positions 24.

The illumination system according to the present disclosure is notlimited to the example described above. For example, the number of thelight source elements and the arrangement pattern of the light sourceelements are not limited to the illustrated examples. An illuminationsystem that includes a plurality of fixed light source elements, such asthe one illustrated in FIG. 9, may include at least four light sourceelements.

The use of such an illumination system C09 makes it possible toimplement the operations described hereinafter.

First, with reference to FIG. 14A, the light source element 20 b that islocated at an initial light source position, among the 25 light sourcepositions, emits the first light, and while the prepared specimen 11 isbeing illuminated with the first light, the image sensor B01 in theprepared specimen 11 captures an image of a sample slice. Data of theimage (first-color image) obtained through the imaging is read out bythe imager 90 from the image sensor B01 and is transmitted to thecontrolling computer C06. This operation for obtaining data of thefirst-color image is repeated while changing the position of the lightsource element that emits light.

Subsequently, with reference to FIG. 14B, in this example, the lightsource element 20 a located at the center of the 25 light sourcepositions emits the first light, and while the prepared specimen 11 isbeing illuminated with the first light, the image sensor B01 in theprepared specimen 11 captures an image of the sample slice. Data of theimage (first-color image) obtained through the imaging is read out bythe imager 90 from the image sensor B01 and is transmitted to thecontrolling computer C06. In addition, the light source element 20 aemits the second light, and while the prepared specimen 11 is beingilluminated with the second light, the image sensor B01 in the preparedspecimen 11 captures an image of the sample slice. Data of the image(second-color image) obtained through the imaging is read out by theimager 90 from the image sensor B01 and is transmitted to thecontrolling computer C06. Furthermore, the light source element 20 aemits the third light, and while the prepared specimen 11 is beingilluminated with the third light, the image sensor B01 in the preparedspecimen 11 captures an image of the sample slice. Data of the image(third-color image) obtained through the imaging is read out by theimager 90 from the image sensor B01 and is transmitted to thecontrolling computer C06.

Subsequently, with reference to FIG. 14C, the light source element 20 blocated at a final light source position, among the 25 light sourcepositions, emits the first light, and while the prepared specimen 11 isbeing illuminated with the first light, the image sensor B01 in theprepared specimen 11 captures an image of the sample slice. As describedabove, data of the image (first-color image) obtained through theimaging is read out by the imager 90 from the image sensor B01 and istransmitted to the controlling computer C06.

The order in which the light source elements 20 emit light is not fixed.The light source element 20 a located at the center may first emitlight, and a first-color image, a second-color image, and a third-colorimage may be obtained. It is not necessary to obtain a second-colorimage and a third-color image successively after a first-color image isobtained while the light source element 20 a located at the center isemitting light. In addition, the position of the light source element 20a that emits light of the three colors does not need to fall at thecenter of the illumination system C09, or the number of the light sourceelements that emit light of the three colors is not limited to one.

Subsequently, with reference to FIGS. 15A and 15B, an embodiment inwhich one or more light source elements are movably supported isdescribed.

In this embodiment, the light source element 22 is moved to multiplelight source positions, and the prepared specimen 11 is illuminated withthe illumination light successively from the respective light sourcepositions. If the exposure time necessary for imaging is sufficientlyshort, the light source element 22 does not need to be paused, and theimaging may be carried out while the light source element 22 is beingmoved. In this embodiment, for example, upon the light source element 22reaching right above the prepared specimen 11, the light source element22 illuminates a subject in the prepared specimen 11 successively withthe first light through the third light having mutually differentwavelengths, and images of the subject are captured. In this case, it ispreferable that the light source element 22 be paused right above theprepared specimen 11.

Subsequently, the image forming apparatus and the image forming methodaccording to an embodiment of the present disclosure are described infurther detail.

FIG. 16A is an illustration for describing a method for capturing abiological image at a magnification of ×2 while changing theillumination angle of the illumination light. FIG. 16A illustrates astate in which the stained slice is illuminated with the illuminationlight from the right above the stained slice.

A light source H01 illuminates a prepared specimen E01 of the CIS methodwith illumination light H02 from the right above the prepared specimenE01. The light source H01 is disposed at a location that is sufficientlyspaced apart from the prepared specimen E01 with respect to the size ofthe prepared specimen 11. Thus, the illumination light H02 can beregarded as parallel light. An optical system that collimates the lightemitted by the light source to produce parallel light may be disposed inan optical path. In that case, the light source can be disposed closerto the prepared specimen E01.

The image sensor B01 illustrated in FIG. 16A includes a semiconductorsubstrate H03, photodiodes H04, a wiring layer H05, light-blockinglayers H06 that cover the wiring layer H05, a transparent layer H07 thatcovers a side of the semiconductor substrate H03 on which light isincident. Portions of the illumination light H02 that have passedthrough a stained slice A05 and are incident on the photodiodes H04 aresubjected to photoelectric conversion by the photodiodes H04 so as togenerate electric charges that constitute an image signal. Theconfiguration of the image sensor B01 is not limited to the exampleillustrated in FIG. 16A.

FIG. 16B is a plan view illustrating a pixel region in the image sensorB01. As illustrated in FIG. 16B, the area of a photodiode (PD) includedin a pixel region is smaller than the area of the pixel region. Here,the area of a pixel region is determined by a product of the pixel pitchin the horizontal direction and the pixel pitch in the verticaldirection. In this example, the ratio (numerical aperture) of the areaof the photodiode (PD) to the area of the pixel region is approximately25%. The imaging surface of the image sensor B01 is covered with thelight-blocking layers H06 except in regions where the photodiodes (PD)are provided. In this example, the image sensor B01 does not include amicrolens array that is for increasing the numerical aperture. Accordingto the embodiment of the present disclosure, as the numerical apertureis smaller, a higher resolution can be achieved.

In the illumination state illustrated in FIG. 16A, light that has passedthrough a region H08, a region H09, and a region H10 of the stainedslice A05 is incident on the corresponding photodiodes H04. The signallevel of the photodiode H04 located right below the region H08 isdetermined by the density of the region H08. In a similar manner, thesignal level of the photodiode H04 located right below the region H09 isdetermined by the density of the region H09, and the signal level of thephotodiode H04 located right below the region H10 is determined by thedensity of the region H10. The density of a region in the presentspecification refers to the optical density (OC) of a region in astained slice. Thus, the magnitudes of the signal charges generated bythe three photodiodes H04 illustrated in FIG. 16A correspond to theoptical transmittances of the region H08, the region H09, and the regionH10.

Meanwhile, a portion of the illumination light that is incident on thelight-blocking layer H06 is irrelevant to the photoelectric conversionby the photodiodes H04. Therefore, a portion of the light that haspassed through the stained slice A05 and is incident on thelight-blocking layer H06 is not reflected on an image signal outputtedby the image sensor B01. In the example illustrated in FIG. 16A, lightthat has passed through either of a region H11 and a region H12 is notreflected on the image signal.

In the example illustrated in FIG. 16A, the width of the photodiode H04along a section parallel to the paper plane is equal to the width of thelight-blocking layer H06. Here, the image sensor B01 is a planar areasensor in which the photodiodes H04 are arrayed in a matrix on atwo-dimensional plane. Therefore, as illustrated in FIG. 17A, regions ofthe stained slice A05 through which the illumination light passes to beincident on the photodiodes H04 are discretely located in the row andcolumn directions. As illustrated in FIG. 16A, the region H08, theregion H09, and the region H10 are located right above the photodiodesH04. Therefore, the densities of these regions determine the signallevels of the corresponding photodiodes H04, which are then detected bythe image sensor B01 in the form of the pixel values in a biologicalimage. In this case, the region H11 and the region H12 are located rightabove the light-blocking layers H06, and thus light that has passedthrough either of the region H11 and the region H12 is not reflected onthe pixel values in the biological image.

What has been described about the region H08, the region H09, and theregion H10 also applies to a region I02, a region I03, and a region I04.A biological image I01 (see FIG. 17B) captured while light source H01emits light is an image that is formed by the pixel values that indicatethe densities or the optical transmittances of the region H08, theregion H09, the region H10, the region I02, the region I03, and theregion I04.

Now, an attention is paid to a pixel 106 illustrated in FIG. 17A. Aportion of the illumination light H02 that has passed through the regionH09, among the region H09, the region H12, a region H14, and a regionH17, of the stained slice A05 is incident on the photodiode H04 withinthe pixel 106. In the pixel 106, light that has passed through any ofthe region H12, the region H14, and the region H17, which are located inthe vicinity of the region H09, is blocked by the light-blocking layersH06. As the image sensor B01 includes the light-blocking layers H06, itbecomes possible to capture an image of a region (sub-pixel region)having a size that is one-fourth the size of the pixel 106.

In the present embodiment, a G-light source that emits G-color light(typically, green light), a B-light source that emits B-color light(typically, blue light), and an R-light source that emits R-color light(typically, red light) are disposed at the position of the light sourceH01. Therefore, the light source H01 can illuminate the subjectsuccessively with the G-color light, the B-color light, and the R-colorlight, serving as the illumination light H02, in the illuminationdirection illustrated in FIG. 16A. Thus, images are captured while thesubject is being illuminated with the illumination light H02 of therespective colors, and three images (G image, B image, and R image) areobtained.

FIG. 18 illustrates a state in which the prepared specimen E01 of theCIS method is illuminated with illumination light in an illuminationdirection different from the illumination direction illustrated in FIG.16A and a biological image is captured. In the example illustrated inFIG. 18, illumination light J02 emitted by a light source J01 isincident on the stained slice A05 at an angle from an upper right sideof the stained slice A05. The light source J01 is disposed at a locationthat is sufficiently spaced apart from the prepared specimen E01 of theCIS method with respect to the size of the prepared specimen 11, andthus the illumination light J02 can be regarded as parallel light. Thelight source J01 according to the present embodiment is a G-light sourcethat emits G-color light. In the description to follow (FIGS. 18, 20,22), the illumination light with which the subject is illuminated in anillumination direction different from the illumination directionillustrated in FIG. 16A is G-color light.

In the illumination state illustrated in FIG. 18, light that has passedthrough the region H11 and the region H12 of the stained slice A05 isincident on the corresponding photodiodes H04. The density of the regionH11 determines the signal level of the photodiode H04 located underneaththe left side of the region H11 in FIG. 18. In a similar manner, thedensity of the region H12 determines the signal level of the photodiodeH04 located underneath the left side of the region H12 in FIG. 18. Inother words, the signal charges generated by the respective photodiodesH04 located to the left in FIG. 18 correspond to the opticaltransmittances of the region H11 and the region H12.

Meanwhile, a portion of the illumination light that is incident on thelight-blocking layer H06 is irrelevant to the photoelectric conversionby the photodiodes H04, and is thus not reflected on the image signaloutputted from the image sensor B01. In the illumination stateillustrated in FIG. 18, portions of the illumination light that havepassed through the region H08, the region H09, and the region H10 arenot reflected on the image signal.

Among the regions illustrated in FIG. 19A, regions through which lightthat is incident on the photodiodes H04 passes are the regions H11 andH12 and regions K02 and K03, which are unconnected regions arrayed inthe vertical and horizontal directions. In other words, when one sees asection orthogonal to the Y-axis in FIG. 19A, a photodiode H04 islocated underneath the left side of each of the regions H11, H12, K02,and K03. Then, the densities of the respective regions determine theoutput levels of the photodiodes H04, and the output levels determinethe pixel values in the biological image. Meanwhile, light that haspassed through any of the region H08, the region H09, and the region H10is incident on the light-blocking layer H06, and is thus not reflectedon the pixel values in the biological image. Therefore, a biologicalimage K01 (see FIG. 19B) captured while the light source J01 is emittinglight is an image formed by the pixel values corresponding to thedensities or the optical transmittances of the region H11, the regionH12, the region K02, and the region K03.

An attention is paid to a pixel K04 illustrated in FIG. 19A. In thepixel K04, a portion of the illumination light J02 that has passedthrough the region H12 is incident on the photodiode H04. Portions ofthe illumination light J02 that have passed through the region H09, theregion H17, and the region H14 included in the pixel K04 are all blockedby the light-blocking layers H06. As the image sensor B01 includes thelight-blocking layers H06, it becomes possible to capture an image of aregion having a size that is one-fourth the size of the pixel.

FIG. 20 illustrates a state in which the prepared specimen E01 of theCIS method is illuminated with illumination light in an illuminationdirection different from the illumination directions illustrated inFIGS. 16A and 18 and a biological image is captured. In the exampleillustrated in FIG. 20, the illumination light J02 emitted by the lightsource J01 that has been moved in the direction of the Y-axis isincident on the stained slice A05 at an angle from an upper right sideof the stained slice A05. The light source J01 is disposed at a locationthat is sufficiently spaced apart from the prepared specimen E01 of theCIS method with respect to the size of the prepared specimen E01, andthus the illumination light J02 can be regarded as parallel light.

In the illumination state illustrated in FIG. 18, the light source ismoved in the direction of the X-axis. Meanwhile, in the illuminationstate illustrated in FIG. 20, the light source is moved in the directionof the Y-axis. Thus, images of a region H13, the region H14, and aregion H15 of the stained slice are captured. Portions of theillumination light J02 that have passed through the region H09 and theregion I03 are incident on the light-blocking layers H06, and are thusnot used in imaging.

Among the regions illustrated in FIG. 21A, regions through which lightthat is incident on the photodiodes H04 has passed are regions L02, L03,and L04, the regions H13, H14, and H15, and regions L05, L06, and L07,which are unconnected regions arrayed in the vertical and horizontaldirections. The densities of the aforementioned regions determine theoutput levels of the respective photodiodes H04, and form the pixelvalues in the biological image. Meanwhile, light that has passed throughthe region H08, H09, H10, H11, H12, or H17 is incident on thelight-blocking layer H06, and is thus not reflected on the pixel valuesin the biological image. Therefore, a biological image L01 (see FIG.21B) captured while the light source J01 is emitting light at theposition illustrated in FIG. 20 is an image formed by the pixel valuescorresponding to the densities or the optical transmittances of theregions L02, L03, L04, H13, H14, H15, L05, L06, and L07.

When an attention is paid to a pixel L08 illustrated in FIG. 21A, aportion of the illumination light J02 that has passed through the regionH14 is incident on the photodiode H04. Portions of the illuminationlight J02 that have passed through the region H09, the region H17, andthe region H12 of the pixel L08 are blocked by the light-blocking layersH06. As the image sensor B01 includes the light-blocking layers H06, itbecomes possible to capture an image of a region having a size that isone-fourth the size of the pixel.

FIG. 22 illustrates a state in which the illumination light J02 emittedby the light source J01 that has been moved in the direction of a linedividing an angle formed by the X-axis and the Y-axis into equal parts(here, the direction of the bisectrix) from the position of the lightsource H01 is obliquely incident on the stained slice A05. Asillustrated in FIG. 22, portions of the illumination light J02 that havepassed through the region H17 and a region H18 are incident on thephotodiodes H04. Meanwhile, portions of the illumination light J02 thathave passed through the region H09 and the region I04 are blocked by thelight-blocking layers H06.

Among the regions illustrated in FIG. 23A, regions through which lightthat is incident on the photodiodes H04 have passed through are a regionH16, the regions H17 and H18, and regions M02, M03, and M04, which areunconnected regions arrayed in the vertical and horizontal directions.The densities of the aforementioned regions determine the output levelsof the respective photodiodes H04, and form the pixel values in thebiological image. Meanwhile, light that has passed through the regionH08, H09, H10, H11, H12, L03, H14, L06, or I04 is incident on thelight-blocking layer H06, and is thus not reflected on the pixel valuesin the biological image. Therefore, a biological image M01 (see FIG.23B) captured while the light source J01 is emitting light at theposition illustrated in FIG. 22 is an image formed by the pixel valuescorresponding to the densities or the optical transmittances of theregions H16, H17, H18, M02, M03, and M04.

When an attention is paid to a pixel M05 illustrated in FIG. 23A, aportion of the illumination light J02 that has passed through the regionH17 is incident on the photodiode H04. Portions of the illuminationlight J02 that have passed through the region H09, the region H12, andthe region H14 of the pixel M05 are blocked by the light-blocking layersH06. As the image sensor B01 includes the light-blocking layers H06, itbecomes possible to capture an image of a region having a size smallerthan the size of the pixel. Therefore, as the numerical aperture issmaller, a higher resolution can be achieved.

Through such procedures, the pixel values corresponding to the densitiesor the transmittances of the respective regions H09, H12, H14, and H17included in the pixel K04 illustrated in FIG. 19A, for example, can beobtained. By combining multiple images obtained in illumination stateswith different illumination directions, a high-resolution image having aresolution that is higher than the resolution of each of theaforementioned images can be generated. In the embodiment of the presentdisclosure, multiple G images obtained while a subject is beingilluminated with G-color light are subjected to the processing describedabove, and another G image having a resolution that is higher than theresolution of each of the aforementioned G images is generated. Thishigh-resolution G image is combined with one or both of one or more Rimages obtained while the subject is being illuminated with R-colorlight and one or more B images obtained while the subject is beingilluminated with B-color light, and thus a high-resolution color image(image that is not monochrome) can be formed.

Instead of varying the illumination angle of the illumination light bychanging the position of the light source that emits light, theillumination angle of the illumination light may be varied by changingthe angle and/or the position of the prepared specimen. Alternatively,the illumination angle of the illumination light may be varied bychanging both the position of the light source that emits light and theangle of the prepared specimen.

FIG. 24A is a block diagram illustrating an exemplary configuration ofthe controller and the image processor according to the presentembodiment.

In the configuration illustrated in FIG. 24A, the imager 90 iselectrically connected to an image sensor (not illustrated) that isdisposed so that light that has passed through the sample slice isincident on the image sensor. This image sensor may be mounted on theprepared specimen as described above, or may be mounted on the imager90.

The controller 120 is connected to the imager 90 and the illuminationsystem C09, and controls the imager 90 and the illumination system C09.The illumination system C09 in the configuration illustrated in FIG. 24Aincludes a G-light source that emits G-color light, a B-light sourcethat emits B-color light, and an R-light source that emits R-colorlight. The illumination system C09 is configured as described above. Thecontroller 120 is further connected to the image processor 140. Theimage processor 140 includes a G-image combiner 142, a B-imageinterpolator 144, and an R-image interpolator 146. The controller 120and the image processor 140 may be implemented by a single computersystem.

In the present embodiment, through the operation of the controller 120,the sample slice is illuminated with the G-color light, serving as theillumination light, successively in different illumination directions,and multiple G images are obtained by the image sensor while the sampleslice is being illuminated with the G-color light. The data of the Gimages is transmitted to the G-image combiner 142 of the image processor140. In addition, the sample slice is illuminated with the B-colorlight, serving as the illumination light, in a single direction, and asingle B image is obtained by the image sensor while the sample slice isbeing illuminated with the B-color light. The data of the B image istransmitted to the B-image interpolator 144 of the image processor 140.Furthermore, the sample slice is illuminated with the R-color light,serving as the illumination light, in a single direction, and a single Rimage is obtained by the image sensor while the sample slice is beingilluminated with the R-color light. The data of the R image istransmitted to the R-image interpolator 146 of the image processor 140.

The G-image combiner 142 combines the G images to generate ahigh-resolution image having a resolution that is higher than theresolution of each of the G images. For example, if the number of pixelsin each G image is 2000×1000, a high-resolution G image having 4000×2000pixels is obtained by combining four of the G images.

The G-image combiner 142, for example, generates a combined image byusing the pixel values included in the biological image I01 (see FIG.17B), the pixel values included in the biological image K01 (see FIG.19B), the pixel values included in the biological image L01 (see FIG.21B), and the pixel values included in the biological image M01 (seeFIG. 23B).

The G-image combiner 142 may determine where in the combined image thepixel values included in the respective biological images are applied.In this case, the G-image combiner 142 may use the positionalinformation of the illumination light when each of the biological imageshas been captured (see FIG. 24B, in which only some of the pixels areillustrated).

The controller 120 may instruct the illumination system C09 as to whichdirection the illumination system C09 should move (the direction in theX-axis or the Y-axis from an initial position, or the direction of aline dividing an angle formed by the X-axis and the Y-axis into equalparts from the initial position) and how much to move (includingpositive and negative values). The illumination system C09 may move inaccordance with the instruction.

The controller 120 may instruct the illumination system C09 to belocated at the predetermined initial position of the illumination systemC09. In this case, the movement amount may be zero. The illuminationsystem C09 may move in accordance with the instruction from thecontroller.

The positional information of the illumination light may be the movementdirection and the movement amount in and by which the controller 120instructs the illumination system C09 to move, or may be the initialposition. Instead of the movement amount, sign information (positive ornegative) indicated by the movement amount may be used.

The B-image interpolator 144 interpolates the received data of the Bimage to increase the number of pixels, and generates an image having anincreased number of pixels. This pixel interpolation, for example, isimplemented by equally providing each pixel value in the original imageto pixels of 2 rows by 2 columns. Therefore, the resolution remainsunchanged even when the number of pixels increases four-fold. Forexample, if the number of pixels in a single B image is 2000×1000,another B image having 4000×2000 pixels is obtained by subjecting thesingle B image to the pixel interpolation. The resolution of the B imageobtained in this manner is unchanged. The method for the pixelinterpolation is not limited to this example.

The R-image interpolator 146 interpolates the received data of the Rimage to increase the number of pixels, and generates an image having anincreased number of pixels. This pixel interpolation is similar to thepixel interpolation implemented by the B-image interpolator 144, and isimplemented by equally providing each pixel value in the original imageto pixels of 2 rows by 2 columns. For example, if the number of pixelsin a single R image is 2000×1000, another R image having 4000×2000pixels is obtained by subjecting the single R image to the pixelinterpolation.

An image output unit 150 receives data of the G, B, and R imagesgenerated, respectively, by the G-image combiner 142, the B-imageinterpolator 144, and the R-image interpolator 145, and outputs a colorimage.

FIG. 25B illustrates an exemplary high-resolution image obtained in thepresent embodiment. Meanwhile, FIGS. 25A and 25C each illustrate ahigh-resolution image according to a comparative example. FIG. 25Aillustrates a high-resolution image generated from a high-resolution Bimage obtained by illuminating a subject with B-color light successivelyin four different directions and combining four captured images, a Gimage obtained by subjecting a single image captured while the subjectis being illuminated with G-color light in one direction to the pixelinterpolation, and an R image obtained by subjecting a single imagecaptured while the subject is being illuminated with R-color light inone direction to the pixel interpolation. In addition, FIG. 25Cillustrates another high-resolution image generated from ahigh-resolution R image obtained by illuminating a subject with R-colorlight successively in four different directions and combining fourcaptured images, a G image obtained by subjecting a single imagecaptured while the subject is being illuminated with G-color light inone direction to the pixel interpolation, and an B image obtained bysubjecting a single image captured while the subject is beingilluminated with B-color light in one direction to the pixelinterpolation.

The comparison between the image illustrated in FIG. 25B and the imagesillustrated in FIGS. 25A and 25C clearly reveals that a combined imagehaving a high resolution as a whole is obtained when the resolution ofthe G image is high. Therefore, it is preferable that the wavelength oflight (first light) with which a subject is illuminated in a largernumber of illumination directions falls in the range of green (from 495nm to 570 nm inclusive).

FIG. 26 is a timing chart illustrating an exemplary operation of anapparatus having a configuration illustrated in FIG. 24A. In the exampleillustrated in FIG. 26, a subject is illuminated successively withR-color light, G-color light, and B-color light in a certain direction,and images are captured at respective instances of illumination.Subsequently, the subject is illuminated with G-color light successivelyin different directions, and images are captured at respective instancesof illumination.

Multiple G images obtained through the imaging are transmitted to theG-image combiner 142. The R image and the B image are transmitted,respectively, to the R-image interpolator 146 and the B-imageinterpolator 144. As can be seen clearly from FIG. 26, the number ofinstances of imaging while the subject is illuminated with the R-colorlight or the B-color light is reduced as compared with the number ofinstances of imaging while the subject is illuminated with the G-colorlight, and thus the total number of instances of imaging is reduced.

Timings at which the R image and the B image are obtained are not fixed,and are not limited to the example illustrated in FIG. 26. In addition,the number of the G images to be obtained is not limited to the exampleillustrated in FIG. 26. Furthermore, the prepared specimen does not needto be an electronic prepared specimen configured as described above, inorder to obtain the effect of the embodiment of the present disclosure.

A variety of super-resolution techniques may be employed to combine themultiple G images obtained while the subject is being illuminated withthe illumination light in different directions into a high-resolutionimage. For example, if an operational expression (matrix) forassociating G images obtained through imaging with a targethigh-resolution image is known, a combined image can be obtained fromimages obtained through imaging by an inverse operation (inversematrix). Such an operational expression depends on the pixel structureof the image sensor and the illumination angle of the illuminationlight, and can be obtained through geometrical optics, through anexperiment, or through a simulation.

Second Embodiment

As described with reference to FIG. 8, the computer C06 may be a devicethat is separate from the imager 90 and/or the illumination system C09.FIG. 27 schematically illustrates an example of an image forming systemconfigured in this manner. An image forming system 100 illustrated inFIG. 27 includes the imager 90, the illumination system C09, and acomputer C08. In the configuration of the image forming system 100illustrated in FIG. 27, an image acquisition apparatus 10A and thecomputer C08 are interconnected. The image acquisition apparatus 10Aincludes the imager 90, the illumination system C09, and the controller120 that controls the imager 90 and the illumination system C09. Thecomputer C08 includes the image processor 140 and the memory 145. Theimage acquisition apparatus 10A and the computer C08 are interconnectedthrough a cable or wirelessly. The computer C08 and the imageacquisition apparatus 10A may be disposed at the same location, or thecomputer C08 may be disposed at a location spaced apart from the imageacquisition apparatus 10A. For example, the image acquisition apparatus10A and the computer C08 may be interconnected through a network, suchas the Internet. The computer C08 here is an independent apparatus.Alternatively, the computer C08 may be circuitry, such as a processor,mounted in a device. The computer C08 and the computer C06 describedabove may each be implemented by a general-purpose computer or adedicated computer (or a general-purpose processor or a dedicatedprocessor).

The operation of the image forming system 100 is substantially the sameas the operation of the image forming apparatus 10 described above. Uponthe prepared specimen 11 being loaded on the imager 90, the image sensorB01 in the prepared specimen 11 becomes electrically connected to theimager 90. As the prepared specimen 11 is loaded on the imager 90, theimage sensor B01 is disposed at a position where light that has passedthrough the sample slice A05 (not illustrated in FIG. 27) is incident onthe image sensor B01. The prepared specimen 11 is not an essentialcomponent of the image acquisition apparatus 10A (or the image formingapparatus 10).

The illumination system C09 emits the illumination light successively indifferent illumination directions relative to the sample slice inaccordance with the control of the controller 120. Here, in accordancewith the control of the controller 120, the image sensor B01 obtainsmultiple first-color images while the sample slice is being illuminatedwith the first light. In addition, the image sensor B01 obtains at leastone second-color image while the sample slice is being illuminated withthe second light. The image sensor B01 may further obtain at least onethird-color image while the sample slice is being illuminated with thethird light.

Image signals or image data of the subject (the sample slice in thiscase) obtained by the imager 90 is transmitted to the image processor140 of the computer C08. For example, data of the G images, data of theB image, and data of the R image are transmitted to the image processor140. The image processor 140 carries out the processing described withreference to FIG. 24A. To be more specific, the image processor 140combines the G images to generate another high-resolution G image havinga resolution that is higher than the resolution of each of theaforementioned G images. In addition, the image processor 140interpolates the B image and the R image. The image data used in theaforementioned processing and intermediary data generated in the courseof the processing may be stored temporarily in the memory 145. The imageprocessor 140 generates a color image having a resolution that is higherthan the resolution of each of the aforementioned G images on the basisof the G image that has been processed to increase the resolutionthereof and the B and R images that have been subjected to theinterpolation processing. The generated image is displayed, for example,in the display C07 (see FIG. 8). The generated image may be stored inthe memory 145 or the database 148 (see FIG. 8). A program that recordsthe above-described processes to be executed by the computer C08 isstored, for example, in the memory 145.

In the image forming system 100, the computer C08 may provide aninstruction (command) for an operation of the controller 120. Thecontroller 120 may be an independent apparatus that is separate from theimager 90, the illumination system C09, and the computer C08. The wholeor part of the image processor 140 and/or the controller 120 may beimplemented by a digital signal processor (DSP), an application-specificintegrated circuit (ASIC), an application-specific standard product(ASSP), a field-programmable gate array (FPGA), a microcomputer, or thelike. The image processor 140 and the controller 120 may be separateprocessors, or two or more of the processors may be included in a singleprocessor. The memory 145 may be part of the image processor 140 and/orthe controller 120.

Now, an exemplary method for changing the illumination angle relative tothe sample slice is described.

FIGS. 28 through 30 illustrate exemplary configurations of theillumination system in which the arrangement of the light source elementand the socket C03 can be modified. In the configuration illustrated inFIG. 28, the illumination system C09 includes an illumination angleadjustment mechanism C10 for changing the posture of the socket C03. Inthe example illustrated in FIG. 28, the illumination angle adjustmentmechanism C10 includes a gonio-mechanism C11 that changes the tilt ofthe socket C03 relative to a reference plane (typically, the horizontalplane). In addition, in the example illustrated in FIG. 28, theillumination angle adjustment mechanism C10 includes a rotationmechanism C12 that changes the angle of rotation of the socket C03relative to a reference direction. The change in the posture in thepresent specification includes a change in the tilt relative to thereference plane, a change in the angle of rotation relative to thereference direction, a change in the position relative to a referencepoint, and so on.

The gonio-mechanism C11 and/or the rotation mechanism C12 are operatedin accordance with the control of the controller 120, and thus theposture of the prepared specimen 11 loaded on the socket C03 can bechanged. Through this, at least one of the positions and the directionsof the sample slice A05 (not illustrated in FIG. 28) and the imagesensor B01 can be changed. For example, when the gonio-mechanism C11and/or the rotation mechanism C12 are operated while the light sourceelement is fixed, the illumination angle relative to the sample slicecan be changed. The illumination angle adjustment mechanism C10 mayfurther include a slide mechanism that translates the socket C03. Theillumination angle adjustment mechanism C10 may be a desired combinationof one or more gonio-mechanisms C11, the rotation mechanism C12, and theslide mechanism.

FIGS. 29 and 30 each illustrate an exemplary configuration of theillumination system C09 that includes an illumination angle adjuster foradjusting the angle at which the illumination light is incident on asample slice. In the configuration illustrated in FIG. 29, anillumination angle adjuster C13 switches between on and off of each of aplurality of light source elements 20 disposed at mutually differentpositions in accordance with the control of the controller 120 (notillustrated in FIG. 29). Through this, the illumination light can bemade to be incident on the image sensor B01 (not illustrated in FIG. 29)in the prepared specimen 11 at different angles. In the configurationillustrated in FIG. 30, the illumination angle adjuster C13 changes theposition at which a light source element 22 is lit by moving the lightsource element 22, for example, along a guide rail in accordance withthe control of the controller 120 (not illustrated in FIG. 30). Throughsuch a configuration as well, the illumination light can be made to beincident on the image sensor B01 (not illustrated in FIG. 30) in theprepared specimen 11 at different angles. By making the illuminationlight be incident on the image sensor in the prepared specimen 11 atdifferent angles, the angle of incidence of the illumination light onthe sample slice can be adjusted such that the illumination lightemitted successively in different illumination directions passes througha different portion of the sample slice and is incident on thephotoelectric converters of the image sensor.

As a mechanism for moving the light source element 22, at least one ofthe gonio-mechanism C11, the rotation mechanism C12, and the slidemechanism described above may be employed. The various mechanismsdescribed above may be implemented by using a known mechanism, such as acombination of a ball screw and a stepping motor, or the like.

The present disclosure can be applied, for example, to a specimenmanagement apparatus for managing specimens.

What is claimed is:
 1. An apparatus, comprising: an illuminator thatemits first light having a peak in a first wavelength and coming from afirst direction to a sample at a first time, third light having a peakin the first wavelength and coming from a second direction to the sampleat a second time, and second light having a peak in a second wavelengthand coming from the first direction to the sample at a third time, thefirst time, the second time and the third time being different; and anoutput that outputs an image based on a first image generated inresponse to the first light, a second image generated in response to thesecond light, and a third image generated in response to the thirdlight.
 2. The apparatus according to claim 1, wherein the illuminatordoes not emit fourth light having a peak in the second wavelength rangeand coming from the second direction to the sample, thereby the imagenot being based on a fourth image generated in response to the fourthlight.
 3. A method, comprising: emitting first light having a peak in afirst wavelength and coming from a first direction to a sample at afirst time, third light having a peak in the first wavelength and comingfrom a second direction to the sample at a second time, and second lighthaving a peak in a second wavelength and coming from the first directionto the sample at a third time, the first time, the second time and thethird time being different; and outputting an image based on a firstimage generated in response to the first light, a second image generatedin response to the second light, and a third image generated in responseto the third light.
 4. The method according to claim 3, wherein theemitting does not emit fourth light having a peak in the secondwavelength range and coming from the second direction to the sample,thereby the image not being based on a fourth image generated inresponse to the fourth light.